Contrast Based Image Fusion

ABSTRACT

A system for two color image fusion blending co-registered low light level images in the visible region of the electromagnetic spectrum with thermal infrared images maximizes the information content of the scene by detecting in which of the two image types, IR and visible, there is more structural information and increasing the weight of the pixels in the image type having the most structural information. Additionally, situational awareness is increased by categorizing image information as “scene” or “target” and colorizing the target images to highlight target features when raw IR values are above a predetermined threshold. The system utilizes Red, Green and Blue (RGB) planes to convey different information such that for targets the Red plane is used to colorize regions when raw IR exceeds the predetermined threshold. For scene images, the Green plane provides improved situational awareness due to the above weighted blend of the two image types.

RELATED APPLICATIONS

This application claims rights under 35 USC §119(e) from U.S.Application Ser. No. 61/976,166 filed Apr. 7, 2014, the contents ofwhich are incorporated herein by reference.

STATEMENT OF GOVERNMENT INTEREST

This invention was made with United States Government support underContract No. W91CRB-07-C-0098 awarded by the Department of the Army. TheUnited States Government has certain rights in this invention.

FIELD OF INVENTION

The invention relates to imaging and more particularly to contrast-basedimage fusion.

BACKGROUND OF THE INVENTION

Heretofore considerable work has been done on night vision devices whichcollect energy from scenes in multiple bands and convert the energy toelectrical signals which are digitally processed, fused and presented inreal time as full motion video on a display for viewing by the user. Oneof these systems is a so-called two color system in which infraredimages and visible light images are fused together in the final image.These prior multiband digitally processed fusion techniques are intendedto increase image detail. A need still exists, however, for a way toblend co-registered low visible light level images with thermal infrared(IR) images in a way that maximizes the scene detail, especially in verylow light conditions, in scenes with very bright lights, and in smoke orfog conditions.

Specifically, in the past, infrared light and visible light have beenfused together in a two color image fusion process that blendsco-registered low light level images. In these systems increasedcontrast enhancement is available through a thermal local area contrastenhancement (LACE) algorithm, and is especially useful in low light andin well illuminated scenarios. Like techniques are applied in thevisible light channel. Both of these local area contrast enhancement(LACE) techniques involved histogram preprocessor functions to addcontrast for improved detail. Moreover, a number of noise rejectionfunctions and algorithms were used to correct for nonuniformity relatedto temperature changes and shifts. Additionally, gain correctionalgorithms provided uniformity for each pixel, whereas row noisereduction algorithms normalized the levels of the rows. Further, clusterde-noise algorithms removed flashing out of a family of pixels in lowlight scenarios, whereas optical distortion correction was appliedbetween the co-registered visible light images and the IR images usingtranslation, rotation and magnification. Finally, focal actuatedvergence algorithms were utilized to correct for parallax errors.

All of the above techniques were used to remove noise and otherartifacts prior to being passed to a fusion algorithm to provide aco-registered fused image composed of infrared and visible light images.

However, there is a need for further improvement of the fused image tobe able to emphasize structural content information in the final fusedimage, thus to further improve image detail.

SUMMARY OF THE INVENTION

Embodiments of the present disclosure provide a system and method thatmaximizes information content in an image fusion process that blendsco-registered low light level images in the visible region of theelectromagnetic spectrum with thermal infrared images, said infrared andvisible images constituting two different image types. Brieflydescribed, in architecture, one embodiment of the system, among others,can be implemented as follows. A fusion module detects which of the twoimage types has a greater quantity of structural information andincreases a weight of the pixels in the image type detected to have thegreater quantity of structural information.

The present disclosure can also be viewed as providing methods ofmaximizing information content in an image fusion process by blendingco-registered low light level visible images in a visible region of theelectromagnetic spectrum with thermal infrared images, said infrared andvisible images constituting two different image types. In this regard,one embodiment of such a method, among others, can be broadly summarizedby the following steps: detecting, in a fusion module, which of the twoimage types of the visible images and the infrared images has a greaterquantity of structural information; and increasing a weight of pixels inthe image type detected to have the greater quantity of structuralinformation.

Other systems, methods, features, and advantages of the presentdisclosure will be or become apparent to one with skill in the art uponexamination of the following drawings and detailed description. It isintended that all such additional systems, methods, features, andadvantages be included within this description, be within the scope ofthe present disclosure, and be protected by the accompanying claims.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features of the subject invention will be betterunderstood in connection with the Detailed Description in conjunctionwith Drawings, of which:

FIG. 1 is a block diagram illustrating the preprocessing of visibleimages and infrared images, followed by a technique that emphasizesdetail in the images, in accordance with a first exemplary embodiment ofthe present disclosure;

FIG. 2 is a block diagram showing the utilization of contrast detectionin the low level visible channel and the infrared channel, in which theweight of pixels in a channel having increased contrast is increased,thus to emphasize images having the better structural information, inaccordance with the first exemplary embodiment of the presentdisclosure;

FIG. 3 is a block diagram showing the utilization of the weightingtechnique described in FIG. 2, combined with additional processingtechniques to increase and emphasize detail in both low light and strongillumination situations, in accordance with the first exemplaryembodiment of the present disclosure;

FIG. 4 is an enhanced digital night vision goggle system simplifiedfunctional block diagram, in accordance with the first exemplaryembodiment of the present disclosure;

FIG. 5 is a video processing pipeline diagram for the enhanced digitalnight vision goggle system, in accordance with the first exemplaryembodiment of the present disclosure;

FIG. 6 is an illustration of histogram segmentation, in accordance withthe first exemplary embodiment of the present disclosure;

FIG. 7 is an illustration of compression of dynamic range segments, inaccordance with the first exemplary embodiment of the presentdisclosure;

FIG. 8 is an illustration of expansion of dynamic range segments, inaccordance with the first exemplary embodiment of the presentdisclosure;

FIG. 9 is a histogram pre-processor with IR LACE example image—dark labw/hot objections, in accordance with the first exemplary embodiment ofthe present disclosure;

FIG. 10 is a histogram pre-processor w/IR LACE example image cold sky,in accordance with the first exemplary embodiment of the presentdisclosure;

FIG. 11 is a Green gain function for the IR channel, in accordance withthe first exemplary embodiment of the present disclosure;

FIG. 12 is an IR scale for the Green channel, in accordance with thefirst exemplary embodiment of the present disclosure;

FIG. 13 is a weighting function for the low light level channel, inaccordance with the first exemplary embodiment of the presentdisclosure;

FIG. 14 is a fusion example image—dark lab, in accordance with the firstexemplary embodiment of the present disclosure;

FIG. 15 is a fusion example image—mixed illumination, in accordance withthe first exemplary embodiment of the present disclosure;

FIG. 16 is a fusion example image—dark woods, in accordance with thefirst exemplary embodiment of the present disclosure; and

FIG. 17 is a fusion example image—lighted woods, in accordance with thefirst exemplary embodiment of the present disclosure.

DETAILED DESCRIPTION

A fusion algorithm fuses thermal images and visible light images byutilizing a blending function based on the contrast, or averagedeviation, in each of two channels, namely, a thermal image infraredchannel and a low light visible image channel, and gives more weight tothe channel with the most structural information. In one embodiment, acontrast detector is utilized for each of the two channels, with pixelsin each of the two channels being weighted in accordance with the outputof the contrast detector to increase the weight of the pixels in thechannel having the most contrast, and thus most structural information.

In one embodiment, the images built up in the aforementioned weightingsystem are made available from a Green plane, which in general producesa situational awareness greenish image of the scene such as would beacquired by night vision goggles.

While the above describes a scene mode, in one embodiment there is acolor enhancing target mode for emphasizing targets when the terrain ishighly illuminated. In order to accommodate the color enhancing targetmode, the image displayed is switched from the scene mode to the targetmode when the raw infrared pixel levels are above a predeterminedthreshold level.

Assuming for the moment, that the raw IR is below the threshold, thenwhat is presented and displayed is the scene mode in which blendedvisible/IR emphasizes pixels in the channel having the more structuralinformation.

When the raw infrared signal is above a user defined threshold, then thetarget mode is what is displayed. Here due to color change algorithmsinfrared pixels change from their normal value to orange or red, whereaslow light level visible pixels are shown in the green with a level equalto the low light level, divided by two. These color changes resulted inan image that constitutes the target image, with a fuse multiplexerswitching to the target mode from the scene mode when the raw infraredis above the aforementioned threshold.

The result is that when raw infrared is below a predetermined threshold,what is presented, is the contrast enhanced scene which blends visibleand infrared based on structural information.

However, when the raw infrared exceeds a predetermined threshold, a fusemultiplexer displays the target image, which is the aforementioned colorchanged image. What happens in this case is that target image pops outdue to the coloration.

According to one embodiment, when in the scene mode, the fusionalgorithm selects the blended visible/IR image that emphasizes imageshaving the better structural information. This blended visible/IR imageis based on the average deviation in the Red and Blue planes as computedglobally over the entire image, or can be computed over local sub-imageregions.

More particularly, in addition to enhancement due to the detection ofstructural information in two channels, in one embodiment, color is usedto emphasize a target. The scene mode and target mode are the two videoor image presentation modes and they are selected by determining whetherdetected infrared exceeds a predetermined infrared threshold. In theillustrated embodiment, when the infrared value for pixel exceeds a userdefined threshold, that pixel is considered belonging to a target andthe fuse multiplexer chooses the target mode instead of the scene mode.When the raw infrared is above this predetermined threshold, the targetcolor mode switches on and colors the particular pixel somewhere betweenorange and red depending on how much low light is present. These colorsnever exist in the scene mode. As a result, in high illuminationsituations when the detected raw infrared exceeds the predeterminedthreshold and the system switches to the target mode, the targets aremade to pop out.

Note that in the target mode the visible light is brought in as Green,with the target pixels being somewhere between orange and red. On theother hand, when in the scene mode, scene mode pixels are from thevisible light channel, with any little infrared being presented asBlue-Green. As the raw infrared level increases these pixels become moreand more yellow.

As will be appreciated, the target mode is operative only when the rawinfrared is above the predetermined threshold. Otherwise the scene modeis used, with the test being on a pixel by pixel basis.

For other types of enhancement, and referring now to the scene mode inwhich there is a Red plane, a Blue plane and a Green plane, limitersensure that the color palette that is chosen is realized. In the scenemode, the limiter associated with the Red plane prevents pixels frombecoming red or orange, as this is reserved for the target mode. Thelimiter associated with the Blue plane reduces the color swing acrossvarying levels of light. The minimum in the Blue plane insures that inan area of strong infrared, the visible aspect still will berepresented.

Thus, in the scene mode and as to the limiters, for IR image pixels thatdo not exceed the thermal target threshold, their intensity is reducedto G/2. In the case of visible light pixels, when the IR pixelintensities do not exceed the thermal target threshold, their intensityis, LLL/4 or LLL-IR, whichever is larger, limited to G/2, where G is theintensity of the Green plane pixels and LLL refers to the intensity ofthe low light level Blue plane pixels.

The technique is best described by comparing images fused using thestandard A+B fusion method and the new local contrast-based fusionmethod. In the contrast fusion method, two images are fused together anddisplayed in the Red/Green/Blue (RGB) color planes using the followingscheme. If the raw IR does not exceed the Target Threshold, the Greenplane displays a weighted combination of the pixels from the thermalcamera and low light level camera to increase the weight of images thathave a high structural content. The weightings which are based onstructural content are a function of the average deviation computedwithin each image, either globally or locally, and are designed to addmore weight to the image region with the most structural content, asdefined by the average deviation.

If the raw IR exceeds a predefined threshold (Target Threshold), the Redplane displays a color enhanced thermal camera pixel and the Blue planedisplays a bracketed or limited version of the low light level camerapixel from the Blue plane.

The resulting detail rich image is the result of simply adding thethermal image pixel values with the low light level pixel values in theGreen plane utilizing the above contrast enhancement algorithm involvingdetecting average deviation in the image.

In addition to the detail enhancement associated with the Green plane,because there is a loss of detail in regions of strong light and in darkareas, generating the Green plane by fusing clipped and weightedversions of the thermal pixels in the Red plane with clipped andweighted versions of the low light level pixels in the Blue plane, theeffect of strong light or darkness is eliminated in the final renderedfused image.

Regardless of the other enhancements described above, the finallyrendered image is the result of the new contrast-based fusion methodwhich provides significantly more detail by increasing the weight ofeither the infrared pixels or the visible light pixels for those imageshaving the stronger structural content as measured by average deviation.

In summary, a system for two color image fusion blending co-registeredlow light level images in the visible region of the electromagneticspectrum with thermal infrared images maximizes the information contentby detecting in which of the two image types, IR and visible, there ismore structural information and increasing the weight of the pixels inthe image type having the most structural information. Additionally,situational awareness is increased by categorizing image information as“scene” or “target” and colorizing the target images to highlight targetfeatures when raw IR values are above a predetermined threshold. Thesystem utilizes Red, Green and Blue (RGB) planes to convey differentinformation such that for targets the Red plane is used to colorizeregions when the raw IR exceeds the predetermined threshold. For sceneimages, the Green plane provides improved situational awareness due tothe above weighted blend of the two image types.

FIG. 1 is a block diagram illustrating the preprocessing of visibleimages and infrared images, followed by a technique that emphasizesdetail in the images, in accordance with a first exemplary embodiment ofthe present disclosure. A fusion enhancement system for use in enhancednight vision goggles takes light from a low light visual channel 10, andthermal images from an infrared channel 12 and pre-processes them asillustrated at 14 and 16, after which the subject fusion algorithm isapplied as illustrated at 18 to display at 20 an enhanced image in whichdetail is increased to aid in situational awareness.

FIG. 2 is a block diagram showing the utilization of contrast detectionin the low level visible channel and the infrared channel, in which theweight of pixels in a channel having increased contrast is increased,thus to emphasize images having the better structural information, inaccordance with the first exemplary embodiment of the presentdisclosure. Central to the enhancement of the fusion process describedin FIG. 1 is the enhancement of structural detail in the finallyrendered image. Here visible light 10 and infrared light 12 in twoseparate channels are applied to respective contrast detectors 22 and 24which detect the contrast in the images in each of these channels. Thecontrast is detected in one embodiment utilizing standard deviationtechniques, with images having increased structural information asdetected by the contrast detectors resulting in increased weight shownat 26 and 28, respectively for the two channels. In this way pixels aremultiplied by the increased weights in the visible and infrared channelsand are summed at 30, at which point they are ultimately used to drivedisplay 20 of FIG. 1, after having been coupled to a fuse multiplexer 38of FIG. 3.

The result of so doing is to provide increased weight to those channelshaving increased structural information. The result is the highlightingor enhancement in a combined image of the infrared and visible imagechannels so that what is presented is an image having increasedsharpness and clarity.

FIG. 3 is a block diagram showing the utilization of the weightingtechnique described in FIG. 2, combined with additional processingtechniques to increase and emphasize detail in both low light and strongillumination situations, in accordance with the first exemplaryembodiment of the present disclosure. The weights W_(IR) and W_(VIS),here shown at 32 and 34 generated through the contrast detection systemof FIG. 2, are applied to a Green plane situational awareness module 36which generates a blended visible/IR image based on structuralinformation. As will be seen, the weighted pixels come from infraredchannel 12 and low light visible channel 10, with the light in thesechannels having been preprocessed as illustrated at 14 and 16 bysophisticated preprocessing techniques to be described hereinafter.

Limiters 42 and 44, having as inputs the preprocessed raw IR fromchannel 12 and the preprocessed raw low light visible light from channel10 process the infrared and visible light and couple them respectivelyto a Red plane 50 and a Blue plane 52. In the case of the Red plane,limiter 42 limits the Red plane pixels to the level associated with theGreen plane pixels divided by two. For Blue plane 52 the visible lightis the greater of LLL/4 or LLL-IR, limited to the Green plane pixelsdivided by two. It will be noted that the limiters ensure that the colorpalette chosen is realized. The limiter on the red channel preventspixels from becoming red or orange, as this is reserved for the targetmode to be described hereinafter. The minimum for the blue channelensures that in an area of strong IR, the visible aspect is stillrepresented.

The result of having generated the Red plane, the Green plane and theBlue plane is that the corresponding scene images 58 are coupled todisplay 20 through fuse multiplexer 38 if the raw intensity of the rawIR is below a predetermined threshold. Thus, in the case where there islow IR, the blended visible/IR scene image 58 based on structuralinformation is used to provide increased clarity and sharpness.

As will be appreciated, what is coupled to display 20 is a fusion firstand foremost of the weighted versions of the visible and infraredchannels. It will be appreciated that the weighted blended visible/IRpixels in the Green plane may be used by themselves to drive display 20.

However, for situations in which the intensity of the raw infraredenergy is above a predetermined threshold 54, fuse multiplexer 38deselects the scene image 58 and selects a target image 62 which is theresult of a color change operation provided by a color change module 60.It will be noted that the inputs to color change module 60 are thepreprocessed infrared light from infrared channel 12 and thepre-processed visible light from low light visible channel 10.

The color change engendered by the color change module is such that theinfrared color is the color red, whereas the green is LLL/2. In thiscase B=0, with the target 62 defined to be the output of the colorchange module.

In operation, when the raw infrared light intensities are less than thethreshold set by threshold detector 54, the scene image 58 is that whichis coupled by the fuse multiplexer 38 to display 20. On the other hand,if the intensity of the raw infrared intensity is greater than thethreshold set by threshold detector 54, then fuse multiplexer 38 selectsthe target image 62 to be coupled to display 20.

The result for low IR is that the blended visible/IR image based onstructural information is coupled to display 20, whereas in situationswhere the raw infrared intensity is greater than the threshold set bythreshold detector 54, it is the color changed image which is coupled byfuse multiplexer 38 to display 20.

Thus, for low light situations, one has increased sharpness based on theweighting of the infrared or visible light depending on which channelhas more structural information, whereas for highly lit scenarios, thatwhich is presented by display 20 is a colorized version which highlightsor pops up targets within the field of view of the cameras.

As described above, the scene or target modes are determined by whetherthe IR exceeds the predetermined threshold. The RGB from both the sceneand the target images are coupled to the fuse multiplexer which controlswhich of the two modes are used based on raw IR levels. Thus, themultiplexer is used to select between scene and target modes based onthe predetermined threshold. It will be noted that when the raw IR valueexceeds a user configured threshold, that pixel is considered a target,and the fuse multiplexer chooses the target mode instead of the scenemode. When the raw IR is above the threshold, the color switches makingthat pixel emphasized as being somewhere between orange and red. Thesecolors never exist in the scene mode.

In the scene mode if there is little IR which will be given a blue greentint. As IR increases, the IR pixels will become more and more yellow.

In summary, structural information is detected in each of the twochannels and pixels having the better structural information are givengreater weights. Secondly, if the infrared channel value is above apredetermined threshold, then color change algorithms enhance the colorsto promote target awareness. Finally, limiters are provided to limit theRed plane and Blue plane components in low light situations and toprevent coloration in case the IR is below the predetermined threshold.The limiter on the red channel prevents pixels from becoming red ororange, as this is reserved for the target mode. The minimum for theblue channel ensures that in an area of strong IR, the visible aspect isstill represented.

FIG. 4 is an enhanced digital night vision goggle system simplifiedfunctional block diagram, in accordance with the first exemplaryembodiment of the present disclosure. In one embodiment, the presentinvention is used in an Enhanced digital night vision goggle system, orEnhanced Night Vision Goggle (digital) ENVG (digital) system, that ishelmet mounted, battery powered, and uses a monocular Night VisionGoggle (NVG) that collects energy from the scene in multiple bands,converts this energy to electrical signals which are digitallyprocessed, fused and presented in real time as full motion video on adisplay for viewing by the user. The enhanced digital night visiongoggle is intended to provide man portable vision capability suitable toperform dismounted and mounted military missions in all terrains, underall light and visibility conditions.

As can be seen in FIG. 4, the night vision goggle system is housed in animage system housing module 70, which includes a thermal objective lensassembly 72 and visible light lens assembly 74 coupled respectively to athermal sensor module 76 and a low light level sensor module 78. Theoutputs of modules 76 and 78 are applied to system electronics 80 thatincludes image fusion, power conversion, digital zoom electronics, andan LED control. Modules 76, 78 and 80 constitute core electronics 81 forthe subject system. The output of system electronics 80 is coupled to amicro display 82 which in one embodiment is a 1280×1024 pixel displayhaving a 24-bit RGB capability which is also gamma corrected. Microdisplay 82 is viewed by an eyepiece lens assembly 84 such that theresult of the image fusion is visible by the naked eye 86. Systemelectronics 80 also includes a land warrior interface 88 and usercontrols 90, as illustrated.

The enhanced digital night vision goggle forms imagery from scene energyin the following bands at a minimum:

-   -   Visible-Near Infrared (VisNIR) 600-900 nm also referred to as        Low Light Level (LLL) herein. Primarily using reflected light        energy from night sky illumination or artificial sources.    -   Long Wave Infrared (LWIR) [8-12 μm] also referred to as Thermal        or Thermal Infrared (TIR) herein. It primarily uses emitted        infrared energy of scene objects.

The system provides a unity magnification, wide Field-Of View (FOV),high resolution, continuous, full motion, and video image of the scene.The system permits the operator to select viewing either of the twobands or the fused product of the two bands. The system is compatiblewith 820-860 nm laser illuminators and pointers (e.g., AN/PEQ-2 andAN/PAQ-4).

Video Processing Pipeline

FIG. 5 is a video processing pipeline diagram for the enhanced digitalnight vision goggle system, in accordance with the first exemplaryembodiment of the present disclosure.

The enhanced digital night vision goggle video processing pipeline isdepicted in FIG. 5, where it can be seen that there is a low light imagepipeline and an IR image pipeline. The outputs of these outlines arecoupled to a combined image focal actuated vergence parallax correctionmodule 73.

As to the low light image pipeline, incoming light is corrected forfixed pattern noise and constant pixel non-uniformity. Furthercorrection is provided by offset and gain sensor mapping module 75. Alsoinvolved is an automatic gain module 77. The outputs of modules 75 and77 are coupled to a cluster de-noise module 79, with modules 75, 77 and79, constituting a noise preprocessor 83. It is noted that the clusterde-noise module 79 removes flashing out of the family of pixels in thelow light image channel. The output of noise preprocessor 83 is appliedto LL LACE module 82 which adds an amount of contrast using histogramtechniques. This concentrates on low light level local area contrastenhancement. The output of LL LACE module 82 is coupled to one input ofmodule 73 used in combined image generation.

As to the IR channel, a fine map module 84 is used to correct fornon-uniformity related to temperature change or shifts. The output ofmodule 84 is coupled to a gain module 86, which corrects fornon-uniformity for each pixel. Module 86 is coupled to row noiserejection module 88, which is used for row noise reduction bynormalizing the levels of the rows. This unit is coupled to a histogrampreprocessor 90 which does a piecewise linear stretch to spread out mostfrequent intensity values within segments, and provides more uniformdistribution of intensity across the histogram. The output of histogrampreprocessor 90 is coupled to IR LACE module 92 which enhances IR localarea contrast by pulling out detail from shadows and highlights. Theoutput of IR LACE module 92 is applied to optical distortion correctionmodule 94, in which optical distortion correction between the low lightchannel and the infrared channel is removed by translation, rotation andmagnification. The output of optical distortion correction module 94 isapplied to the other input of module 73. The combined image which is theoutput of module 73 is applied to a fusion module 96 so as to providepreprocessed low light image information and preprocessed IR imageinformation.

More particularly, the functionality of the processing pipelines is nowdescribed in more detail:

Low Light Pipeline: NUC FPN

The NUC FPN 75 processing function includes the offset and gain sensormapping. The offset map corrects for fixed pattern noise including readnoise, noise associated with background and dark current. The offset mapwill also correct for constant pixel non-uniformities. The gain mapcorrects for pixel response non-uniformity as well as non-uniformityrelated to the lens. Through the gain operation one allows bit depth togrow to 16 bits.

Low Light Pipeline: NUC AutoGain

The AutoGain module 77 controls the high voltage power supply on the LLLsensor. The major input to this control is light level which isdetermined through the mean of the LLL image. The high voltage dutycycle is then adjusted with a PID loop so that the best possibleresponse can be achieved. The AutoGain module also includes sensorprotection from saturation or damage.

Low Light Pipeline: Cluster De-Noise

The Cluster De-Noise module 79 addresses flashing out of family pixelsthat are prevalent at very low light levels. A rank order filter is usedto determine outlying pixels and they are then filtered out. This is aproven technique used in the visible camera industry to reduce noise.

Low Light Pipeline: LL LACE

LL LACE module 82 is primarily adding an amount of contrast in theimage. The block operates on both a global and local level to enhancethe contrast in the image. This operation is performed using a filterkernel and a global histogram stretch. The image is reduced to 8 bitsduring this operation.

Thermal Pipeline: Fine Map

In the IR image pipeline, the fine map module 84 processing involves athermal calibration done using the system shutter. The goal of the finemap is to correct for non-uniformity related to temperature change orsystem drift. Given the extreme sensitivity of the thermal sensor thismap is required to be adjusted as the system operates.

Thermal Pipeline: Gain Map

The Gain Map processing shown at 86 is a thermal calibration done duringsystem build. The gain operation corrects for response non-uniformityfor each pixel. The bit depth of the image goes from 14 bit to 16 bit inthis step. The gain map also has the ability to substitute and replaceunresponsive pixels.

Thermal Pipeline: RNR

The RNR (Row Noise Reduction) module 88 processing is an algorithm thatnormalizes the level of the rows locally.

Thermal Pipeline: Histogram Preprocessor

The Histogram Preprocessor module 90 is essentially a Piecewise LinearStretch of the histogram of the incoming image. In this approach, thehistogram of the incoming image is divided into eight (8) segments, andeach segment is stretched using a linear function to spread out the moredensely populated sections of the histogram over the entire segment.This essentially provides a more uniform distribution of intensityacross the histogram, in preparation for contrast enhancement.

Thermal Pipeline: IR LACE

IR LACE module 92 is primarily for adjusting the amount of contrast inthe image. The block operates on both a global and local level toenhance the contrast, pulling out detail from the shadows andhighlights. This operation is performed using a filter kernel and aglobal histogram stretch. The image is reduced to 8 bits during thisoperation.

Thermal Pipeline: ODC+

The optical distortion correction module 94 processing corrects formechanical and optical differences between the LLL subsystem and the IRsubsystem. Corrections include translation, rotation, magnification anddistortion mismatch.

Combined Pipeline: FAV

The FAV (Focal Actuated Vergence) module 73 processing contains atechnique employed to correct for parallax errors between the LLL and IRsubsystems. This algorithm adjusts the vertical offset on the thermalimage based on the focus point of the LLL lens. This provides properalignment for any part of the image that is in focus.

Combined Pipeline: Fusion

Within the enhanced digital night vision goggle video processingpipeline, the Fusion algorithm in fusion module 96 provides TargetCueing (TC) and Situational Awareness (SA) under all weather andillumination conditions utilizing numerically efficient methods chosento provide low-SWAP and low latency.

The Fusion algorithm utilizes metrics provided by the low light level(LLL) and thermal pipelines to adapt to dynamic scenes. The contrastdetector located in LLL LACE utilizes several Signal to Noise Ratio(SNR) metrics to determine how much to rely on the LLL and the thermalsensors for the situation awareness channel. In relatively high lightconditions, almost the entire situational awareness image is mapped fromthe LLL sensor. As lighting conditions deteriorate, the LLL SNRdecreases, the LLL sensor contribution is decreased and thermal sensordata fills the gap maintaining high situational awareness resolution andcapability.

What is now discussed is the algorithm utilized for the histogrampreprocessor utilized in the IR channel.

Histogram Pre-Processor (Piecewise Dynamic Range Reduction)

An algorithm used for the histogram pre-processing of the IR channelbefore local area contrast enhancement (LACE) is applied. The purpose ofthis preprocessing is both to analyze the distribution of the pixelvalues in the image for use in contrast enhancement and also to scalethe distribution to reduce problems associated with large dynamic rangedistributions. Such conditions can be due to very hot objects or regionswith large differences in average temperature, such as a warm forest inthe foreground with a cold sky in the background. These conditions areknown as bimodal distributions due the separation of the histogram forsuch images into two distinct Gaussian-like distributions, often with alarge gap between them. Proper adjustment of the dynamic range of theimage in such cases permits effective processing by subsequentalgorithms to properly enhance the fine detail in the different regionsand prevents over-saturating the values in bright regions and washingout darker regions.

To this end a piecewise histogram scaling method compresses regions ofthe distribution that are sparsely populated, such as the gaps betweenthe distributions in a bimodal case, and expands regions that aredensely populated. Additionally, the method is designed to restrict theexpansion of dynamic range values so as to minimize the amplification ofnoise and creation of artifacts, which is a weakness of the standardhistogram-based global contrast enhancement techniques such as plateauequalization. The technique also has the advantage that it does notexhibit large variations in illumination as the brightness of the scenevaries and it does not create significant flickering in video sequencescompared to other histogram based techniques. All the same, it retainsthe overall simplicity and a low burden of processing requirementscommonly associated with such global methods.

Algorithm Details

The algorithm involves four major steps in the processing:

1. Histogram Development—First, the histogram of the image is calculatedusing 4 k bins over the entire 16-bit range of possible values in theimage. This histogram actually comes from the previous frame in thehardware implementation due to the low-latency requirements of thegoggle.

2. Segmentation of the Histogram—Next, the distribution is segmentedinto K points which determine the illumination levels at discrete valuesof the population. The distribution of these points gives an indicationof the dynamic range regions which are either densely or sparselypopulated, thereby indicating where one must compress the dynamic rangeand where one must expand the dynamic range.

3. Dynamic Range Specification—Then, the lengths of each segment areused to determine whether compression or expansion of the dynamic rangeis used for that interval. Based on that, the value of the length ofeach segment for the final image distribution is determined by a simplehistogram specification procedure. From this, the offset values andscaling coefficient for each interval can be calculated.

4. Piecewise Scaling—Finally, the values of each of the pixels in theimage are adjusted by a scaling procedure which uses the offset valuesand scaling coefficients.

Histogram Development

First, the histogram of the image is calculated using 4 k bins over theentire 16-bit range of possible values in the image. This histogramactually comes from the previous frame in the hardware implementationdue to the low-latency requirements of the goggle. The requirement fornumber of bins comes from the fact that one needs to accuratelycharacterize the distribution for the scaling procedure, without overlyburdening the processing requirements. Generally speaking, a properlyimaged region will have a dynamic range on the order of 512 values, sothe quantization of the values by 16 results in roughly 32 bins oversuch regions, which provides sufficient resolution of the variousdynamic-range regions to allow for accurate segmentation and scaling ofthe values. Also, for one implementation one uses only a bin every4^(th) pixel to reduce the memory requirement of the histogram buffer toan acceptable level.

Segmentation of the Histogram

Next, the distribution is segmented into K (equals 32 for theimplementation) points which determine the illumination levels atdiscrete values of the population. The segmentation of the dynamic rangedistribution is based upon finding the discrete values in the dynamicrange that correspond to certain predetermined values of the pixelpopulation. To this end, one first creates the cumulative distributionfunction from the histogram in the usual way. Next, one determines thesegmentation points using the following equation:

$\begin{matrix}{{{CDF}\left( L_{k}^{l} \right)} = {{\sum\limits_{i = l_{\min}}^{L_{k}}{H(i)}} = C_{k}^{l}}} & {{Equation}\mspace{14mu} 1}\end{matrix}$

In this equation, the k^(th) segmentation value L_(k) ^(t) is determinedas the value for which the CDF equals

k^(th) threshold value C_(k) ^(t). The threshold value is defined by:

C _(k) ^(t) =F _(k) ^(t) ×CDF(L _(max))  Equation 2

The distribution of the threshold factors _(k) is a configuration tablefor the current implementation, F^(t), and was and was originally chosento be a linearly spaced set of values from 0 to 1. However, it was foundthat a logarithmically distributed set of values worked better.

An important issue is the determination of the extreme values of theimage distribution to decide how to clip the minimum and maximum valuesto reduce the effects of outliers on the overall scaling of theillumination and brightness. This is determined by the values of thefirst and last thresholds, which were set to:

F _(lower) ^(clip)=0.005,F _(upper) ^(clip)=0.98   Equation 3

These values were based on thresholds used for previous versions of ourcontrast enhancement algorithms that worked well for the enhanceddigital night vision goggle.

The current implementation uses 32 segment points, which provides adecent balance between simplicity and effectiveness. Originally, 8values were used which worked well enough to prove the concept butresulted in unacceptable amounts of artifacts due to the coarseness ofthe scaling. An illustration of this segmentation for 8 points is givenin FIG. 6.

FIG. 6 is an illustration of histogram segmentation, in accordance withthe first exemplary embodiment of the present disclosure. This sort ofdistribution is typical for cases where a hot object such as a light isin the image, and in fact the long tail is typically several orders ofmagnitude larger than the distribution of the “hump” which contains mostof the interesting image detail. Simple clipping of such an image willnot solve the problem due to the fact that the hot object may occupy asignificant proportion of the image.

Now that one has the segmentation of the image distribution, it isnecessary to decide how the distribution should be scaled. This isaccomplished by first determining the values of the distribution pointsfor the corrected image.

Dynamic Range Specification

The next step in the process is to determine the scaling and offsets foreach interval. This is accomplished by mapping the dynamic rangesegments identified by the segmentation procedure onto a “canonicalgrid” which represents some ideal distribution. This is actually a formof histogram specification, although the specification used was somewhatheuristic and was chosen based on observation of the distributions of avariety of well-formed IR images which did not have dynamic rangeissues. Another issue that influences the specification was the desireto not over enhance the values of the dynamic range at lowertemperatures, which typically results in amplification of undesirablespatial noise.

FIG. 7 is an illustration of compression of dynamic range segments, inaccordance with the first exemplary embodiment of the presentdisclosure. FIG. 8 is an illustration of expansion of dynamic rangesegments, in accordance with the first exemplary embodiment of thepresent disclosure. The most important contribution of the algorithm isthe compression of large dynamical range segments that have little or noimage information, which is illustrated in FIG. 7. The interval ΔL_(k)^(t) is the difference between the levels L_(k) ^(t) and L_(k-1) ^(t)which were shown in FIG. 6. For example shown here, the k^(th) segmentis larger than the canonical segment ΔL_(k) ^(c). The scaling of thisinterval based on the ratio of the 2 intervals is also determined, aswell as described in more detail below. First, what is explained is howto handle the opposite case for which the original segment is smallerthan the canonical segment as illustrated in FIG. 8.

In this case the condition would suggest expansion of the interval.However, experience with this was mostly negative, as this frequentlyleads to amplification of spatial noise in the image, while yieldingonly modest benefits in terms of contrast enhancement (in general).Thus, it was decided for this version of the algorithm to just leave thesegment length alone for this case, as the improvements due to dynamicrange compression was the pressing issue.

The result of this step of processing is simply the choice of the finalscaled interval size for each segment. This can be summarized by theformula:

ΔL _(k) ^(s) =L _(k) ^(s) −L _(k-1) ^(s)=min(ΔL _(k) ^(t) ,ΔL _(k)^(c))   Equation 4

In this formula ΔL_(k) ^(s) is the value of the final scaled interval,ΔL_(k) ^(t) is the value of the un-scaled, original interval found bythe segmentation procedure, and ΔL_(k) ^(c) is the value of the“canonical” or pre-specified segment interval which is a configurationvalue stored in a look-up table for the hardware implementation

Piecewise Scaling

Once one has the segments of the original distribution and values forthe final dynamic range intervals, the calculation of the offsets andscaling coefficients for each segment is based on simple linear scaling.The formula for this is given by:

F _(x,y) ^(s) =S _(k)(F _(x,y) ^(u) −L _(k) ^(t))+L _(k) ^(s)   Equation5

Here, F_(x,y) ^(s) is the final scaled pixel value, F_(x,y) ^(u) is theoriginal un-scaled value, L_(k) ^(t) is the value of the segment pointthat is just less than the original pixel value, and L_(k) ^(s) is thenew offset for that segment. The scaling factor is given by the ratio ofthe scaled to un-scaled interval for that segment.

$\begin{matrix}{S_{k} = \frac{L_{k}^{2} - L_{k - 1}^{s}}{L_{k}^{t} - L_{k - 1}^{t}}} & {{Equation}\mspace{14mu} 6}\end{matrix}$

The scaled offsets are simply the accumulation of the values of theinterval sizes for all of the segments below that segment:

$\begin{matrix}{L_{k}^{s} = {\sum\limits_{j = 0}^{j = {k - 1}}{\Delta \; L_{k}^{s}}}} & {{Equation}\mspace{14mu} 7}\end{matrix}$

The value of the first offset is arbitrary, and is chosen to be thevalue of the lower clip value of the original distribution forsimplicity. Another obvious choice would be to set it equal to zero.

Imagery Examples

Several example images comparing the previous goggle algorithm with thenew algorithm are shown below. These examples show how the method solvesa couple of important problems for large dynamic range imagery.

FIG. 9 is a histogram pre-processor with IR LACE example image—dark labw/hot objections, in accordance with the first exemplary embodiment ofthe present disclosure. FIG. 10 is a histogram pre-processor w/IR LACEexample image cold sky, in accordance with the first exemplaryembodiment of the present disclosure. The first example shows how thealgorithm solves the problem of very hot objects in an image causingfade in the detail for the rest of the image, thereby masking otherobjects such as a person. The original contrast enhancement algorithmdid not handle this case well as shown by the image collected from thegoggle on the left of FIG. 9, while the right hand side shows the newpre-processing algorithm implementation.

Another important case is that of a warm foreground with a coldbackground. Here, what is shown is an image of a scene with interestingobjects which is degraded due to the cold sky causing contrast reductionin the warmer foreground (left-hand side of FIG. 10), and the same scenewith the preprocessor active (right-hand side of FIG. 10).

As illustrated in this case, many details that were completely washedout are now visible, and the person is clearly visible.

Context-Based Fusion, Weighting Based on Structural Information

Different smoothly varying blends of low light level (LLL) images andthermal (IR) images were evaluated in order to create a fused image forthe goggle. The objective of the investigation was to address severaluser-identified deficiencies in the existing algorithm—primarily toprovide more thermal in areas where the visible contrast is low. Lowvisible contrast primarily happens when the light level is very low(inside dense foliage, buildings, or tunnels) or when there is smoke orfog. The final blending algorithm was a function of the contrast foreach image type, where contrast is defined as the standard deviation ofthe image. Blending gives more weight to the image type with the moststructural information as measured by the standard deviation of thatimage.

Algorithm Approach

The subject system produces two image modalities (visible and thermal)and has three color planes (Red, Green, and Blue) to display theinformation. Using the theory of opponent colors, the fused image colorscheme was designed to provide specific information regarding the scenebased on colorization. Constraints were placed on the image colorscheme, including:

-   -   The primary situation awareness information should be shades of        Green or hues close to Green.    -   Red colors are reserved for “hot” thermal targets.

Based on these constraints, the Green channel was designated to containthe primary situation awareness information. Soldiers trained in the useof photon-intensified low light cameras are used to seeing a Greenimage. Therefore, subject goggle should provide familiar images that areprimarily Green, but enhanced with Red to indicate targets, yellow toindicate higher levels of thermal, and Blues to indicate higher levelsof visible light. In order to accomplish this, the Green channel iscomposed of a blend of visible and thermal imagery. The Red channelcontains the thermal image with each pixel limited to one half of theGreen pixel level, providing shades of Green to yellow. The Blue channelshows the greater of excess visible over the thermal or one quarter ofvisible with each pixel limited to one half of the Green pixel level,thus providing shades of Blue. The reason Blue is an excess level is toprevent the colors from all being white when thermal and visible levelsare similar. This coloring scheme provides the following perceptualinformation:

-   -   If thermal values exceed the IR threshold, the pixels are        colored Red to indicate a “hot” target. The pixel values are        shades of Red in low levels of visible light and turn        orange-yellow as the visible light level increases.    -   In very low light levels, the Green channel is mainly composed        of enhanced thermal combined with a fraction of the visible        light image (minimize the speckle while still showing strong        lasers).    -   As the visible light level increases, Green is composed of a        blend of the visible image mixed with a fraction of the thermal        image.    -   The Red channel contains the thermal image and each pixel level        is bounded by G/2. Yellow in the fused image indicates areas        where the thermal image is stronger than the visible image.    -   The Blue channel is the excess of visible over thermal and        bounded by <LL/4 . . . G/2>. Blue or purple in the fused image        indicates areas where the visible image is much stronger than        the thermal image.

Algorithm Details

The primary algorithm effort involved finding a smooth function to blendvisible images with thermal in the Green channel based on theirrespective contrast levels. Since standard deviation is directly relatedto contrast, the robust average deviation of the image was calculated:

$\begin{matrix}{{ADEV} = {\frac{1}{N}{\sum\limits_{i = 1}^{N}{{{x(i)} - µ}}}}} & {{Equation}\mspace{14mu} 8}\end{matrix}$

where N is the number of pixels, x(i) is the ith pixel level, and μ isthe mean value of the image. In one embodiment, the average deviationwas calculated for both the pre-LACE visible and pre-LACE thermalimages. In order to calculate the average deviation for the thermalimage with dimensions 640×480 and not using any divides, only 640×410pixels were used and result was shifted by 18 (divided by 512*512):

$\begin{matrix}{{ADEV}_{IR} = {\frac{1}{512*512}{\sum\limits_{i = 36}^{445}{\sum\limits_{j = 1}^{640}{{{I_{IR}\left( {i,j} \right)} - µ_{IR}}}}}}} & {{Equation}\mspace{14mu} 9}\end{matrix}$

A Matlab code segment to calculate the mean and average deviation forthe visible images is shown in Exhibit 1.

Exhibit 1: Matlab code for average deviation of visible image % AverageDeviation Code for LL Nr = 1024; % Number of rows in LL image Nc = 1280;% Number of columns in LL image nBitShiftLL = −20; % Divide by 1024*1024% Compute the average value of LL NUC mLL = 0; for i = 1:Nr % for eachrow for j = 1:Nc % For each pixel in each col mLL = mLL + LL(i,j); endend mLL = bitshift(mLL, nBitShiftLL); % Mean LL RNR % Compute theaverage deviation of LL NUC sLL = 0; for i = 1:Nr % for each row for j =1:Nc % For each pixel in each col sLL = sLL + abs(LL(i,j)−mLL); end endsLL = bitshift(sLL, nBitShiftLL); % ADEV LL NUCA Matlab code segment to calculate the mean and average deviation forthe thermal images is shown in Exhibit 2.

Exhibit 2: Matlab code for average deviation of thermal image % AverageDeviation Code for IR Nr1 = 36; % Start row in IR RNR image NR2 = 445; %End row in IR RNR image NcIR = 640; % Number of columns in IR imagenBitShiftIR = −18; % Divide by 512*512 % Compute the average value of IRRNR mIR = 0; for i = Nr1:Nr2 % for each row for j = 1:NcIR % For eachpixel in each col mIR = mIR + IR(i,j); end end mIR =bitshift(mIR,nBitShiftNR); % Mean IR RNR % Compute the average deviationof IR RNR sIR = 0; for i = Nr1:Nr2 % for each row for j = 1:NcIR % Foreach pixel in each col sIR = sIR + abs(IR(i,j)−mIR); end end sIR =bitshift(sIR,nBitShiftIR); % ADEV IR RNRContrast-based fusion uses the estimates of the average deviations toblend visible and thermal images into the Green channel. The completefusion algorithm using these estimates is shown in the table in Exhibit3.

Exhibit 3: Matlab code for fusion % Contrast-Based Fusion Algorithm ifIR > Thresh_sig % IR threat  R = I_(IRL)  G = I_(LLL)/2  B = 0 else % inbackground G = min(W_(LL)*I_(LLL) + W_(IR)*I_(IRL),255) R =min(mI_(IRL),.5*G) B = min(max((I_(LLL)−I_(IRL),0.25*I_(LLL)),.5*G) endWhere W_(LL) = min(m_(LL)/16+.5,1) W_(G) =min(max(64/(s_(IR)+s_(LL)),.3),1.25) W_(IR) =W_(G)*s_(IR)/(s_(IR)+s_(LL)) s_(IR) = IR average deviation s_(LL) = LLaverage deviation m_(LL) = LL mean valueIf the raw thermal pixels exceed a user-controlled threshold, thosepixels are colored a shade of Red. The Red channel is set to the thermalimage intensity, while the Green channel is set to half the intensity ofthe visible image in which targets get an orange hue in strong visiblelight.

Images with thermal pixel values less than the target threshold providesituation awareness, with the primary information in the Green channel.The Red channel provides yellow hues where the thermal image is high,while the Blue channel provides Blues and purples in regions wherevisible intensities exceed thermal intensities. The function for theGreen channel enhances the thermal a little and diminishes visible alittle when visible contrast is low. The following gain function wasused to weight the thermal image based on the contrast of the twoimages:

W _(G)=min(max(64/(s _(IR) +s _(LL)),0.3),1.25)   Equation 10

The resulting values are placed in a lookup table and shifted to theleft by 24 samples:

WgLUT=min(max(64./(1:256),0.3),1.25);   Equation 11

WgLUT(1:233)=WgLUT(24:256)   Equation 12

FIG. 11 is a Green gain function for the IR channel, in accordance withthe first exemplary embodiment of the present disclosure, and shows aplot of this function.

The thermal weighting function for the Green blend is:

W _(IR) =W _(G) *s _(IR)/(s _(IR) +s _(LL))   Equation 13

The complete lookup table implementation of this algorithm is calculatedas shown in Exhibit 4.

Exhibit 4: Matlab code to develop lookup table for IR Green GainFunction % Compute the green gain function iG = max (sLL+sIR,1) % indexmust be at least 1 iG = min(iG,128); % maximum index range is 128 WgLUT= min(max(64./(1:256),.3),1.25); WgLUT(1:233) = WgLUT(24:256); Wg =ggLUT(iG); % Compute the IP weighting function D = max (sLL+sIR,1); %Denom must be at least 1 D = min(D,1024); % Force range to a maximumvalue Wir = Wg*sIR/D; % One mult and one div

FIG. 12 is an IR scale for the Green channel, in accordance with thefirst exemplary embodiment of the present disclosure. The resulting IRweighting for various levels of visible contrast are shown in FIG. 12.The visible image is gradually reduced in intensity when the light levelis very low. The purpose of this weighting function is to reduce imagespeckle where there is little information, but still allow strong laserpointers or spot lights to be clearly seen.

The weighting function for the visible image in the Green channel is afunction of the mean value of the image:

W _(LL)=min(m _(LL)/16+0.5,1)

This weighting function is shown in FIG. 13, which depicts a weightingfunction for the low light level channel, in accordance with the firstexemplary embodiment of the present disclosure.

Imagery Examples

Several example images comparing the previous goggle algorithm with thenew algorithm are shown below. These examples show the corner cases ofthe algorithm. The first example in FIG. 14, which is a fusion exampleimage—dark lab, in accordance with the first exemplary embodiment of thepresent disclosure, is a very low light level image taken in a darkroomwith the following statistics:

-   -   μLL=0.72, σLL=5, σIR=19

The original goggle algorithm on the left has little thermal, has lotsof visible speckle, and a strong light under the door. The new algorithmon the right provides a good blend of enhanced IR, reduces the visiblespeckle, but preserves the strong light under the door.

The next example of FIG. 15, which is a fusion example image—mixedillumination, in accordance with the first exemplary embodiment of thepresent disclosure, shows the strong thermal being blended in to theimage even when there is relatively strong visible. In this case, thebright visible lights are still evident around the porch light and theheadlights, but the details in the thermal background are also broughtout in shades of yellow and Green. The image statistics are:

-   -   μLL=65, σLL=96, σIR=254

The following example of FIG. 16, which is a fusion example image—darkwoods, in accordance with the first exemplary embodiment of the presentdisclosure, was taken in the woods and the visible light wasartificially lowered. The scene is primarily thermal, but there is astrong laser light in the middle left. Contrast fusion preserves theGreen channel for situation awareness and uses shades of yellow toindicate strong thermal in the image. The statistics for the images are:

-   -   μLL=0.3, σLL=1, σIR=56

The same scene in much strong light is shown in FIG. 17, which is afusion example image—lighted woods, in accordance with the firstexemplary embodiment of the present disclosure. The image is primarilyvisible, but the thermal is fused in at a lower level. The imagestatistics are:

-   -   μLL=175, σLL=83, σIR=56

It will be appreciated that the new contrast-based fusion algorithm usesa blending algorithm that gives more weight to image types (thermal orlow light level) that have the most structural content. The newalgorithm maximizes the scene information content, providing more detailin low contrast areas (inside buildings, caves, or under tunnels), or inwashed out areas (in the presence of strong lights, in smoke, or infog).

While the present invention has been described in connection with thepreferred embodiments of the various Figures, it is to be understoodthat other similar embodiments may be used or modifications andadditions may be made to the described embodiment for performing thesame function of the present invention without deviating there from.Therefore, the present invention should not be limited to any singleembodiment, but rather construed in breadth and scope in accordance withthe recitation of the appended claims.

What is claimed is:
 1. A system that maximizes information content in animage fusion process that blends co-registered low light level images inthe visible region of the electromagnetic spectrum with thermal infraredimages, said infrared and visible images constituting two differentimage types, comprising: a fusion module for detecting which of the twoimage types has a greater quantity of structural information andincreasing a weight of the pixels in the image type detected to have thegreater quantity of structural information.
 2. The system of claim 1,wherein the fusion module for detecting which of the two image types hasthe greater quantity of structural information includes contrastdetectors for each of the two image types that detect a contrast for thetwo image types.
 3. The system of claim 2, wherein said contrastdetectors include modules for determining an average standard deviationof pixel values for pixels in an image.
 4. The system of claim 3,wherein the average standard deviation is taken over an entire image. 5.The system of claim 3, wherein the average standard deviation is takenover a predetermined region within an image.
 6. The system of claim 1,further comprising a camera having infrared and visible outputstherefrom, wherein said fusion module detects the structural informationin said infrared and visible outputs.
 7. The system of claim 6, furthercomprising preprocessing modules for each of said infrared and visibleoutputs.
 8. The system of claim 6, further comprising a display, whereinan information content of said infrared and visible outputs is madeavailable to said display in terms of a Red plane, a Green plane, and aBlue plane.
 9. The system of claim 8, wherein images from said Greenplane are displayed at said display for situational awareness of a sceneviewed by said camera.
 10. The system of claim 8, wherein images in saidRed plane, said Green plane and said Blue plane are categorized as beingeither one of: scene images and target images.
 11. The system of claim10, wherein scene images are displayed when raw infrared values fromsaid camera are below a predetermined threshold.
 12. The system of claim10, wherein target images are colorized when raw infrared values fromsaid camera are above a predetermined threshold.
 13. The system of claim12, wherein the color of infrared images is colorized so as to changethe color from a normal color associated with the infrared image to animage which has an increased red color, wherein the increased red coloremphasizes target features of a target when said raw infrared levels areabove said predetermined threshold.
 14. The system of claim 13, whereinsaid target features are colored red to orange, thus to emphasize saidtarget features.
 15. The system of claim 8, and further including afirst limiter coupled to said infrared image for limiting the value ofinfrared pixels to be no greater than the value of Green plane pixelsdivided by two.
 16. The system of claim 8, wherein said visible imagehas visible image pixels and further includes a second limiter coupledto said visible image setting a value of said visible image pixels tothe greater of the low light level image value divided by at least oneof four and the low light level image value, minus a raw infrared value,whichever is greater, and limiting to the value of the Green planepixels divided by two.
 17. The system of claim 1, wherein situationalawareness is increased by categorizing image information as at least oneof scene information and target information and colorizing the targetimages to highlight target features when raw IR values are above apredetermined threshold.
 18. The system of claim 17, further comprisinga color change module for categorizing said image information as atleast one of scene information and target information based on infraredpixel value inputs.
 19. The system of claim 18, wherein said colorchange module changes infrared pixels to a shade of orange to red.
 20. Amethod of maximizing information content in an image fusion process byblending co-registered low light level visible images in a visibleregion of the electromagnetic spectrum with thermal infrared images,said infrared and visible images constituting two different image types,the method comprising: detecting, in a fusion module, which of the twoimage types of the visible images and the infrared images has a greaterquantity of structural information; and increasing a weight of pixels inthe image type detected to have the greater quantity of structuralinformation.